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ABSTRACT 

This report describes the following components of the 
Nestle Workplace Literacy Project: six job task analyses, curricula 
for six workplace basic skills training programs, delivery of courses 
using these curricula, and evaluation of the process. These six job 
categories were targeted for training: forklift 1 oader/checker , BB 1 s 
processing systems operator, department mechanic, Pearson kitchen 
helper, Butterfinger packaging systems operator, and Butterfinger 
manufacturing systems operator. A total of 109 participants attended 
classes from these target groups. The curriculum development process 
included design of instruction for a 48-hour class in each job 
category. According to the evaluation, four of the six courses 
resulted in observable improvements in job performance. The report 
includes an overview of the training conducted, description of job 
performance outcomes, and discussion of learner achievement outcomes 
for each of the six job categories. Summaries of answers to the 
evaluation questions grouped by the six job categories are appended. 
A critical review of the Internal Evaluation is divided into three 
sections. The first establishes the basis for evaluation review or 
met aevaluat i on and gives those models with differing criteria of 
goodness in evaluation. The second section looks at the structure of 
the evaluation project as a whole. It makes suggestions for 
reformatting the contents and enhancing readability. Section 3 makes 
specific comments on the draft itself. The review closes by endorsing 
the evaluation of what appeared to be an excellent educational 
program. (YLB) 
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EXECUTIVE SUMMARY: 



The Nestle Workplace Literacy Project completed all of its 
activities as outlined in the grant proposal. The components of 
the project included six job task analyses, curricula for six 
workplace basic skills training programs, delivery of courses 
utilizing these curricula, and evaluation of the process. 

The project was completed within the time parameters originally 
contracted and within the budget guidelines established. A 
summary of the final budget expenses is included in this report. 

Four of the six courses resulted in observable improvements in 
job performance. The remaining two courses had perceptions of 
improvement by the trainees, but little documentation by 
supervisors and external observers. One of these was the first 
course offered. The other was the victim of scheduling problems 
as a result of production requirements. These scheduling problems 
negatively affected some of the participants, which led to some 
attendance fluctuations . 

The original objectives were satisfied: 

1. Improvements were recorded for participants as evidenced in a 
decrease in scrap and rework, increased production output, c,nd 
improved quality. The plant -wide safety record also improved. 

2. Improvements in reading, writing, problem solving, and team 
work were noted by most of the trainees. 

3. Approximately 50% of those targeted agreed to attend the 
courses offered. Classes were mandatory for one group of 
employees. All others attended voluntarily. Some participants 
continued to attend other skills classes after completion. 

4. Several marginally performing incumbents improved their skills 
sufficiently as a result of this training program to ensure their 
tenure in their present positions. 

5. Attendance was very good. Several early leavers resulted from 
transportation problems and lack of interest in the subject 
matter, but overall attendance was very good. Most of the classes 
were held right before or right after the normal work shift with 
overtime pay provided for attendance.^ 

6. Instructor quality was rated very high by participants. 

7. Supervisors indicated the project had a very positive effect 
on participants and resulted in less calls to them for assistance 
for minor production difficulties. 
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EXECUTIVE SUMMARY (Cont.) 



8. Trainees indicated a high degree of satisfaction with the 
project . 

9. The fact that the project was successfully implemented is in 
itself an indicator of positive collaboration and teamwork 
between organized labor and management. Other food manufacturers 
who also received Federal Workplace Literacy grant money (e.g. 
Nabisco) have been unable to complete their project as planned 
due to an inability of labor and management to cooperate. 



SUMMARY ANALYSIS: 

The concept of providing basic skills training in a workplace 
context enhanced workers skills. This skill enhancement then 
resulted in improved production quality and quantity. Workers 
attitude toward the learning process improved as a result of 
their participation in the classes. The program was attended by 
half of the target population, which is the largest market 
penetration ever at this site for a basic skills training 
program . 

One of the keys to success for this project was the detailed 
task analysis performed for each of the positions. The original 
project plan did not allow enough time to complete this aspect of 
the project. Actual delivery of instruction was the lowest cost 
professional expense, so some job task analysis time was charged 
to instruction. The curriculum developed as a result of the job 
task analysis included specific lesson plans and class 
activities. This level of detail was critical to the success of 
the project . 

Class scheduling was most effective when classes were held for 
2 -hour periods twice weekly. This seemed to be enough time for 
activities to be conducted without draining the trainees 
energy. The classes that were conducted on 8-hour training 
segments did not show the same positive results, It appears that 
the trainees were not able to concentrate and internalize their 
learning with longer classes. 

Pre and post testing appeared to be useful when the 
participants were stable and the groups were large enough to take 
advantage of ability grouping. For small, heterogenous groups, 
individual educational plans are recommended. 
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RECOMMENDATIONS : 



Based on the finding of this project, future projects 
utilizing the workplace-based method for basic skills training 
should: 

- develop detailed job task analyses for curriculum building 

- build curriculum in a general manner, allowing the individual 
participants to customize material based on their needs. 

- offer classes in 2-hour segments, meeting twice weekly on 
non- consecutive days . 

- design pre and post tests that accurately reflect course 
content . 

- use ability grouping when scheduling classes. 

- provide individual lessons on computer-based media early in 
the program, not later. 

- implement a staff in-service program at the onset of the 
proj ect . 

- consider the benefits of hiring full-time professionals 
instead of consultants on an hourly basis. 

- investigate the possibility of using a full-time project 
director on-site. 

- begin extensive marketing efforts as soon as the target 
populations have been identified. 
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INTRODUCTION : 



This report summarizes the activities of the Workplace Literacy 
Grant funded under award number U198A30096. The project began on 
3/1/93 and was completed on 8/31/94. 

Six job categories were targeted for training through this grant: 
Forklif t Loader/Checker, BB ; s Processing Systems Operator, 
Department Mechanic, Pearson Kitchen Helper, Butterfinger 
Packaging Systems Operator, and Butterfinger Manufacturing 
Systems Operator. A total of 109 participants attended classes 
from these target groups. 

A task analysis was conducted for each of the six job 
categories. The task analysis included observation of skilled 
workers on each job, interviews with workers and supervisors, and 
study of documents used in the workplace. From this task analysis 
a curriculum was developed to help improve the basic skills of 
the workers in the six job categories. The basic skills list was 
adopted from the American Society of Training and Development 
publication, Workplace Basics (1990, Carnevale, Meltzer, and 
Gainer) . 



The curriculum development process included design of instruction 
for a 48 hour class for each of the six job categories. The 
instruction was designed to be very workplace specific in 
context, yet resulting in improved basic skills as an outcome of 
the training. 

In most cases three 48 -hour classes were offered, one for each 
shift of workers in each job category. There were two exceptions 
the forklif t loader/checker group only had one session, since 
all interested participants could be accommodated at one time, 
and the BB's Processing Operators, who met for 6 8-hour full-day 
sessions. The four other job categories met in 2-hour classes 
meeting twice weekly for twelve weeks, with sessions at the 
beginning of the afternoon and evening shifts, and at the end of 
the day shift. 

This report will include a summary of answers to the evaluation 
questions and recommendations for future training. 

The information used to answer the evaluation questions was 
obtained by interviewing course participants and their 
supervisors, examining production data, analyzing class 
documentation, and observing participant behavior. 
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I. FORKLIFT LOADER/ CHECKER 

Overview: 

The forklift loader/checker class was conducted from July through 
October, 1993. A total of 8 loader/checkers were eligible to 
participate. Only 3 chose to do so. One left the class early. The 
class was held on regular work hours for 2 hours, twice per week, 
for a total of 48 hours. Topics covered in the class were based 
on the job task analysis performed earlier that year. The 
curriculum included arithmetic, reading, listening, learning to 
learn, and basic problem solving. All these subjects were taught 
using workplace context examples. 

Job Performance Outcomes : 

Although there is still a need for more training in order to 
lower the error rate, observations by supervisors and by 
evaluators indicated that the loader error rate decreased for 
those who participated in the course. The most positive effect 
observed by a supervisor was an increased effort to achieve 
quality. According to the supervisor, "They (the adult learners) 
seem to grasp the concept that they are not 'just loading a 
truck' they are 'filling a customer order.'" 

Learner Achievement Outcome s : 

Positive knowledge gains were recorded in basic reading, writing, 
and mathematics. 



II. PEARSON KITCHEN HELPER 

Overview: 

The Pearson Kitchen Helper classes were conducted from 
December , 1993 through March, 1994 ♦ A total of 9 Pearson employees 
were eligible to participate; 8 chose to do so. There were no 
early leavers . The class was held either before or after 
regular work hours for 2 hours, twice per week, for a total of 48 
hours. Topics covered in the class were based on the job task 
analysis performed earlier that year. The curriculum included 
arithmetic, reading, listening, learning to learn, and basic 
problem solving. All these subjects were taught using workplace 
context examples. 

Job Performance Outcomes: 

Supervisors reported improved job performance by participants and 
also indicated that participants make fewer requests for 
assistance since taking the course. In addition, scrap and rework 
decreased in areas where participants are assigned and production 
'start-ups' are being accomplished more efficiently 



PEARSON HELPER (Cont.) 



Learner Achievement Outcomes: 

Seventy-five percent (75%) of participants showed significant 
improvement in subject areas taught while 25% sharpened up 
existing competencies, but showed no new learning gain. Several 
participants were motivated to enroll in additional classes to 
further enhance their reading skills. 

Ill, DEPARTMENT MECHANIC 

Overview: 

The Department Mechanic classes were conducted from January 
through March, 1994. A total of 95 maintenance employees were 
eligible to participate; 37 chose to do so. There were 5 early 
leavers . The class was held either before or after regular work 
hours for 2 hours, twice per week, for a total of 48 hours. 
Topics covered in the class were based on the job task analysis 
performed earlier that year. The curriculum included arithmetic, 
algebra, trigonometry, and problem solving. These subjects were 
taught using workplace context examples. 



Job Performance Outcomes: 

Participants exhibited increased speed in performing machine 
repairs after completing course. Participants also evidenced 
greater diagnostic accuracy as a result of the training received. 



Learner Achievement Outcomes: 

Participants showed significant improvement in mathematics and 
problem solving ability. 



IV. BB'S PROCESSING SYSTEMS OPERATOR 

Overview : 

The BB's Processing Systems Operator classes were conducted from 
January through April, 1994. A total of 31 were eligible to 
participate; all chose to do so. Therfe were no early leavers . 
The class was held either during or in addition to regular work 
hours for 8 hours, once per week, for a total of 48 hours. Topics 
covered in the class were based on the job task analysis 
performed earlier that year. The curriculum included arithmetic, 
reading, vocabulary, and problem solving. These subjects were 
taught using workplace context examples. 
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BB'S PROCESSING SYSTEMS OPERATOR (Cont . ) 



Job Performance Outcomes : 

According to supervisors, job performance of participants has 
improved , as evidenced in a reduction of calls for assistance for 
minor process deviations. Total production output for the jobs 
affected has been more consistent , leading to the ability to open 
distribution to additional geographic regions in advance of 
schedule. Several previously marginal performers , who were in 
danger of being put on probation for poor performance, showed the 
.necessary skill improvement to prevent this from happening. 

Learner Achievement Outcomes : 

Participants improved their skill in reading of operator 
interfaces. Teamwork and communication skills were positively 
impacted. 

V. BUTTERFINGER PACKAGING SYSTEMS OPERATOR 

Overview : 

The Butterfinger Packaging Systems Operator classes were 
conducted from April through June, 1994. A total of 36 were 
eligible to participate; 19 chose to do so. There was 1 early 
leaver. The classes were held either before or after regular work 
hours for 2 hours, twice per week, for a total of 48 hours. 
Topics covered in the class were based on the job task analysis 
performed earlier that year. The curriculum included arithmetic, 
reading, vocabulary, and problem solving. These subjects were 
taught using workplace context examples. 

Job Performance Outcomes : 

Supervisors and managers reported improved job performance as 
evidenced in ability to diagnose production problems more quickly 
and more accurately as well as improved quality of daily reports 
as a result of the writing classes. Supervisors also noted 
participants' work had a noticeably higher quality after 
participation in the learning activities. Production rates for 
participants improved while scrap and rework rates declined. 
Supervisors reported fewer calls for advice for minor machine 
problems. Participants were observed y/orking more cooperatively 
with mechanics and electricians working on the production line. 

Learner Achievement Outcomes : 

Participants exhibited improved achievement in mathematics, 
reading, writing, and problem solving. Interpersonal 
communications and teamwork were also improved as a result of the 
learning activities of participants. 
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VI. BUTTERFINGER ' MANUFACTURING SYSTEMS OPERATOR 



Overview : 

The Butterfinger Manuf acturirg Systems Operator classes were 
conducted from April through June, 1994. A total of 12 were 
eligible to participate; 8 chose to do so; 2 additional 
participants attended through their own choice. There were early- 
leaver. The classes were held either before or after regular work 
hours for 2 hours, twice per week, for a total of 48 hours. 
Topics covered in the class were based on the job task analysis 
performed earlier that year. The curriculum included arithmetic, 
reading, vocabulary, and problem solving. These subjects were 
taught using workplace context examples. 

Jc b Performance Outcomes : 

Supervisors reported improved performance on startpups, more 
accurate completion of daily reports, and a reduction in calls to 
supervisors for advice for minor problems. A decline in scrap and 
rework was noted in the participants' area and there was an 
improvement in production output. Supervisors reported fewer 
calls for advice and direction on routine production deviations. 
Also, supervisors noted improved communication with participants 
as a result of the learning activity. 

Learner Achievement Outcomes : 

There wer : achievement increases for participants in mathematics, 
reading, and problem solving. Greater understanding of teamwork 
was attained. 
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BUDGET STATUS 8/11/94 
CATEGORY SPENT TO DATE TOTAL ALLOCATION COMMENTS 



Federal : 
Job/Task 
Assessment 

Curriculum 
Development 

Instruction 



Evaluation 



$39,042.50 
$141,452.87 
$122,517.84 

$8165.79 



$36 , 676 Complete 



$132,430 Complete 



$133, 705 Travel to 

final conf . 

to be billed 
in Nov. 

$7700 Complete 



Admini- 
stration 

Indirect 



$52,315.05 



$18,000 



$50, 000 



$18, 026 



Complete 



Complete 



Total 



$381,494.05 



$378, 537 



Complete 



Cost Sharing : 

Curriculum 
Development 



$14,397.50 



$23,370 



Instruction 

Trainee 
Wages 

Computers 

Total 



$3710.00 

$110,871.03 

$56,607.01 
$185,585.54 



$23,595 

$68,112 

$55, 000 
$178,581 



Matching 

Requirement 

Met 
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APPENDIX B 



I. PORKLIFT LOADER/ CHECKER 

OBJECTIVE NUMBER ONE: CORRELATE WORKPLACE LITERACY SKILLS 
ENHANCEMENT WITH MANUFACTURING PERFORMANCE . 

EVALUATION QUESTIONS: 

1. What are the average and typical outcome measures of workplace 
skills enhancement? 

a) What are average and typical learner reaction outcomes? 
Learners indicated an above average rating on the overall course. 



b) What are average and typical learning outcomes? 

All learners showed slight increases in math, reading, and 
problem solving test scores. Since there were only three learners 
in this group detailed group performance statistics were not 
computed, due to the small group size. 

c) What are average and typical performance-based outcomes? 
Supervisor and learner interviews and evaluator observations 
indicate an improvement in performance. The loading error rate 
has decreased slightly for those who participated in the class 
compared to those who did not participate. Unfortunately, the 
error rate is still considered too high by plant management. 
Attitude toward the job has improved. As the department manager 
stated, "They seem to grasp the concept that they are not just 
loading a truck they are filling a customer order!" 

2. What are key indicators of quality manufacturing performance? 
Load error rate is continually calculated. As mentioned above, 
the error rate is slightly less for those who participated in the 
class, but still not satisfactory. 

3. What is the magnitude of the correlation of these measures? 
The correlation was not calculated due to the small group size. 

OBJECTIVE NUMBER TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES 

EVALUATION QUESTIONS: 

1. Did the training positively impact trainee attitudes toward 
the learning process? 

Yes, in most trainees displayed very positive attitudes toward 
the learning process. At first, there was a lot of peer pressure 
not to participate, but those who decided to enroll became very 
creative in inventing excuses for why they decided to attend. One 
of the learners who had very good reading and math skills wished 
to study investing strategies. This learner was given direction 
in how to pursue a personal learning project to improve 
investment strategies . 
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OBJECTIVE NUMBER-' TWO : IMPROVE WORKPLACE LITERACY SKILLS OF 



TRAINEES (Cont.) 



2 . Did the trainee appreciate the learning experience? 
The trainees all expressed appreciation for the learning 
opportunity, especially during regular working hours. 

3. How did the trainee rate: 

the learning experience average to above average 

the assessment process resistant to testing, but liked the 

individual attent ion . 

course usefulness initial enthusiasm, then waning interest 
course practicality weren't sure why they were being 

taught how to load trucks they do it evey day! 
instructor performance high ratings for instructor 
course content some resistance to basic math instruction 
course activities? in general, good acceptance, liked more 
action exercises, less math drill. 

4. Did the trainee feel (s)he was a partner in the learning 
process? 

Trainees were asked what areas they would like to improve in, and 
the course content was adjusted to meet these requests. Due to 
the charter to teach basic skills required to perform the 
specific tasks needed, this individualization had to be blended 
with the basic skills component. 

5. To what extent were line workers basic skills increased? 
Reading, math, and problem solving skills showed moderate 
improvement. One learner is still in need of intensive basic 
reading and comprehension tutoring. 

6. Were the learning objectives met? 

Learning objectives were set by the curriculum developer and 
modified by the instructor to ir.eet the needs of the individual 
learners. Both learners who completed the course were satisfied 
that their individual objectives were met. 

7. In a pre/posttest comparison, were there positive knowledge 
gains in basic reading, writing, and math skills? 

Yes, positive gains in erll three areas. 

8. Did trainees perceive an acceleration in knowledge gains? 

All three learners had not participated in any formal educational 
programs recently. This course increased their ability to absorb 
new knowledge. 

9 . Was the program long enough? 

The trainees felt the program was too long, yet they are still 
making errors in loading their trucks. Perhaps training is not 
the total answer to the error-rate problem. 
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OBJECTIVE NUMBER' TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES (Cont.) 

10. Did the trainees have ample time to integrate subject 
content, skills, and processes? 

The course was spread over a twelve week period. This appeared to 
be ample time to integrate new learnings. There were no concerns 
raised that this was not possible due to a time constraint. 



OBJECTVE NUMBER THREE: DESIGN AND ESTABLISH PROGRAM WHICH 
ENCOURAGES UTILIZATION. 

EVALUATION QUESTIONS: 

1. What are the average and typical frequencies of use? 
Three participants out of a total of 8 possible yields a 37,5% 
frequency of participation. Plant management had hoped for a much 
higher level of participation since the error rate was so high. 
After completion of the course, the non-participant 
loader/checkers were again approached and asked to attend another 
session, but none were interested. 



OBJECTIVE NUMBER FOUR: QUALIFICATION RATES FOR ADVANCEMENT 
EVALUATION QUESTIONS: 

1. To what extent has the Workplace Literacy Program provided 
opportunities for advancement? 

None at this time. The purpose of the fcrklift loader/checker 
course was to improve truck loading accuracy. There has been no 
side effect influencing opportunities for advancement to date. 



2. To what extent has the program increased the qualification 
rate of job applicants? 

This training program was designed to enhance the skills of 
current job holders hot applicants. Disqualification of the 
participants is possible, but not likely, especially after their 
performance improvement as a result of their class participation. 



3. To what extent do employees feel a x sense of empowerment, added 
responsibility, and increased decision making? 

None measurable at this point. The loader/checker job is designed 
to allow the worker to have complete control over the truck 
loading process. The training program was designed to improve the 
learners skills needed to accurately complete the loading 
process . 
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OBJECTIVE NUMBER/ FIVE: ATTENDANCE 



EVALUATION QUESTIONS: 

1. What was the frequency of attendance? 

Attendance was good, in excess of 95% This may be attributable to 
class being held on regular work hours. 

2. Did anyone leave the program? 

One person left the program early. This learner stated she was 
bored, and that this class wasn't helping her on her job. The 
error rate summary conducted after the class ended dii not 
confirm her assertion that she didn't need this class. Her error 
rate was similar to those who did not attend. 

3. Why did trainees leave the program? 

As stated above, learner was not interested in the course 
content . 



II ♦ PEARSON KITCHEN HELPER 

OBJECTIVE NUMBER ONE: CORRELATE WORKPLACE LITERACY SKILLS 
ENHANCEMENT WITH MANUFACTURING PERFORMANCE. 

EVALUATION QUESTIONS : 

•1 . What are the average and typical outcome measures of workplace 
skills enhancement? 

a) What are average and typical student reaction outcomes? 

The trainees rated the classes from 4 to 5 on a five-point scale, 
with 1 as a low point, 

b) What are average and typical learning outcomes? 

Pre and post test scores were compared. Seventy- five percent of 
the learners demonstrated significant improvements in test score. 
The remaining 25% did not show any improvement, 

c) What are average and typical performance-based outcomes? 
As reported by supervisor interviews, performance improved 
overall. Supervisors are receiving fewer inquiries for help and 
assistance from the trainees for minor process deviation 
corrections , 

2. What are key indicators of quality manufacturing performance? 
Scrap and rework have decreased in the area affected by the 
trainees performance. Production start-ups have been at expected 
rates, Piior to training production start-ups were often below 
expectations , 
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OBJECTIVE NUMBER ONE: CORRELATE WORKPLACE LITERACY SKILLS 
ENHANCEMENT WITH MANUFACTURING PERFORMANCE . (Cont.) 

3. What is the magnitude of the correlation of these measures? 
This data has not been statistically correlated. Interviews with 
supervisory personal have provided data that suggest improved 
performance by a majority of the trainees. 

OBJECTIVE NUMBER TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES 

EVALUATION QUESTIONS: 

1. Did the training positively impact trainee attitudes toward 
the learning process? 

Trainees all indicated that they enjoyed their work more after 
completion of the original training program. Several trainees 
attended additional programs to enhance reading skill. 

2. Did the trainee appreciate the learning experience? 
Trainees expressed strong appreciation for the experience 
verbally to this writer. 

3. How did the trainee rate: 

the learning experience — positive reaction to the course 
the assessment process -~ used as a diagnostic tool, it was 
viewed as helpful. 

course usefulness — learning was applied on the job daily, 
course practicality very positive correlation; actual 
documents and operator interfaces were used in the class, 
instructor performance -- 4.5 on 5-point scale (5 -high) 
course content 4.7 rating 
course activities 4.5 rating 

4. Did the trainee fell (s)he was a partner in the learning 
process? Several of the trainees participated in the course 
design; all trainees were able to influence course content during 
the course delivery. 

5. To what extent were line workers basic skills increased? 
Reading of panelview screens improved, teamwork improved, verbal 
and written communications were positively impacted, and basic 
mathematical skills improved. 

6. Were the learning objectives met? 

All objectives were met to various degrees. 

7. In a pre/posttest comparison, were there positive knowledge 
gains in basic reading, writing, and math skills? 

No testing was done for this class in writing skills. In reading 
and math, 75 % of the trainees improved. 
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OBJECTIVE NUMBER' TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES (Cont.) 

8. Did trainees perceive an acceleration in knowledge gains? 
Yes. All trainees indicated that they were able to learn faster 
toward the end of the clas*~ sessions. 

9. Was the program long enough? 

No . Several trainees have attended additional classes in reading 
to improve their basic skills. 

10. Did the trainees have ample time to integrate subject 
content, skills, and processes? Yes. Courses were scheduled for 2 
hour sessions, twice weekly* The time between classes was 
sufficient to integrate classroom theory into workplace practice. 

OBJECTVE NUMBER THREE: DESIGN AND ESTABLISH PROGRAM WHICH 
ENCOURAGES UTILIZATION. 

EVALUATION QUESTIONS: 

1. What are the average and typical frequencies of use? 
Attendance was very good (95%-f) . In several instances, trainees 
attended classes during vacation periods, even though they were 
not required to come to work during their regular shift. 

OBJECTIVE NUMBER FOUR: QUALIFICATION RATES FOR ADVANCEMENT 

EVALUATION QUESTIONS: 

1 . To what extent has the Workplace Literacy Program provided 
opportunities for advancement? At this time, no advancement has 
been available. This objective has not been met. 

2. To what extent has the program increased the qualification 
rate of job applicants? No new job applicants participated in 
this training program. Incumbents in this position have improved 
their skill level in adapting to new technologies recently 
introduced to their work environment. 

3. To what extent do employees feel a sense of empowerment, added 
responsibility, and increased decision making? Positive impact in 
this area. Employees are making fewer class to supervisors for 
routine decisions. 

OBJECTIVE NUMBER FIVE: ATTENDANCE 
EVALUATION QUESTIONS: 

1. What was the frequency of attendance? 
Above 95%. 
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OBJECTIVE NUMBER'' FIVE: ATTENDANCE (Cont.) 



2 . Did anyone leave the program? 

No early leavers. All continued to completion. 

3 . Why did trainees leave the program? 
Not applicable. 

III. DEPARTMENT MECHANIC 

OBJECTIVE NUMBER ONE: CORRELATE WORKPLACE LITERACY SKILLS 
ENHANCEMENT WITH MANUFACTURING PERFORMANCE. 

EVALUATION QUESTIONS: 

1 . What are the average and typical outcome measures of workplace 
skills enhancement? 

a) What are average and typical student reaction outcomes? 
Overall reaction to the course was rated 4.2 on a 5-point scale 
with 5 being high. 

b) What are average and typical learning outcomes? 

On math and problem-solving tests, the average improvement from 
pre to post test was a 33% improvement in test score. 

c) What are average and typical performance-based outcomes? 
Several of the trainees reported the ability to diagnose and 
machine problems faster and more accurately as a result of their 
participation in this course. Due to the diverse nature of job 
assignments it was not possible to collect data comparing one 
trainee to another on a pre and post training basis. 

2. What are .key indicators of quality manufacturing performance? 
Ability to diagnose and accurately repair machine malfunctions .is 
the primary responsibility of the trainees. As mentioned above 
the speed and accuracy of these repairs has increased in several 
instances . 

3. What is the magnitude of the correlation of these measures? 

A slight decrease in total time to complete a repair (10-15%) . A 
slight increase in accuracy of diagnosis (25% improvement) . 

OBJECTIVE NUMBER TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES 

EVALUATION QUESTIONS: 

1. Did the training positively impact trainee attitudes toward 
the learning process? Yes. Several trainees responded very 
favorably to additional training, both internal and external to 
their workplace. 
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OBJECTIVE NUMBER TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES (Cont.) 

2. Did the trainees appreciate the learning experience? 

The overall course was very positively received in interviews 
with this writer. 

3. How did the trainee rate: 

- the learning experience 4.2 on a 1-5 scale (5 high) 

- the assessment process enjoyed taking the pre- test; 
remarked that it was very challenging. 

- course usefulness course forced participants to use new 
methods of problem solving. This was positively received. 

- course practicality mathematics review was helpful to 
performance of workplace calculations, 

- instructor performance - 4 . 6 on a 1-5 scale, 

- course content - 4.4 on a 1-5 scale, 

- course activities -- 4.1 on a 1-5 scale . 

4. Did the trainee feel (s)he was a partner in the learning 
process? Only two of the trainees were part of the initial 
planning process. After the first few class sessions, a 
consensus -building process was used to ensure more trainee 
participation . 

5. To what extent were line workers basic skills increased? 
Math skills improved by 33%; Problem- solving skills improved by 
32%. as measured by pre and post tests. 

6. Were the learning objectives met? Original plan included 
teaching and practice in elementary algebra and trigonometry. 
This objective was not fully met. All other objectives were 
satisfied. 

7. In a pre/posttest comparison, were there positive knowledge 
gains in basic reading, writing, and math skills? Reading and 
writing were included in this course. Math skills improved by 
33%. 

8. Did trainees perceive an acceleration in knowledge gains? 
Yes, very much so. Several trainees remarked very favorably about 
their ability to learn faster and more efficiently. 

9. ^ Was the program long enough? Additional time could have been 
devoted to pre-work to eliminate the need for review of basic 
arithmetic in the course. This would have provided enough time to 
review the elementary algebra and trigonometry. 

10. Did the trainees have ample time to integrate subject 
content, skills, and processes? The course schedule of 2 hour 
classes held twice weekly provided enough time for application of 
classroom theory to work practice. 
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OBJECTVE NUMBER THREE: DESIGN AND ESTABLISH PROGRAM WHICH 
ENCOURAGES UTILIZATION. 

EVALUATION QUESTIONS: 

1. What are the average and typical frequencies of use? 
Attendance averaged over 90%. 

OBJECTIVE NUMBER FOUR: QUALIFICATION RATES FOR ADVANCEMENT 
EVALUATION QUESTIONS: 

1. To what extent has the Workplace Literacy Program provided 
opportunities for advancement? There were no opportunities for 
advancement at this time . 

2. To what extent has the program increased the qualification 
rate of job applicants? No change in the qualification rate. 

3. To what extent do employees feel a sense of empowerment, added 
responsibility, and increased decision making? Participants 
reported increased confidence in their problem analysis and 
decision making abilities. 

OBJECTIVE NUMBER FIVE: ATTENDANCE 

EVALUATION QUESTIONS: 

1 . What was the frequency of attendance? 
Attendance was very good over 90%. 

2. Did anyone leave the program? 

Five participants out of a total of 37 left the program early. 

3 . Why did trainees leave the program? 

One left due to a time conflict with another certification test; 
the other four were not interested in the course content. 



IV. BB'S PROCESSING SYSTEMS OPERATOR 

OBJECTIVE NUMBER ONE: CORRELATE WORKPLACE LITERACY SKILLS 
ENHANCEMENT WITH MANUFACTURING PERFORMANCE. 

EVALUATION QUESTIONS: 

1. What are the average and typical outcome measures of workplace 
skills enhancement? 

a) What are average and typical student reaction outcomes? 
The course was rated 4 . 1 on a 1-5 scale, with 5 being high. 
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OBJECTIVE NUMBER ONE: CORRELATE WORKPLACE LITERACY SKILLS 



ENHANCEMENT WITH MANUFACTURING PERFORMANCE . (Cont.) 



b) What are average and typical learning outcomes? 
Learners reported an increase in their ability to read the 
operator interface screens used to control their equipment. 
Instructors reported an increase in participants vocabulary and 
recognition of special -use acronyms and abbreviations. 

c) What are average and typical performance-based outcomes? 

On the job performance as rated by the participants supervisors 
has shown some improvement in reduction of calls for assistance 
for minor process deviations. 

2. What are key indicators of quality manufacturing performance? 
Key indicators of quality performance are scrap and rework rates, 
quality control inspections, and production output. 

3 . What is the magnitude of the correlation of these measures? 
Very slight changes in these indicators. Total production output 
for the jobs affected by these training sessions has been more 
consistent, leading to the ability to open distribution to 
additional geographic regions in advance of schedule. 

OBJECTIVE NUMBER TWO; IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES 

EVALUATION QUESTIONS: 

1. Did the training positively impact trainee attitudes toward 
the learning process? As reported by trainee reaction sheets, the 
typical attitude toward the learning process improved through the 
activities of this class. 

2 . Did the trainee appreciate the learning experience? 

The trainees did appreciate the experience. It allowed them the 
opportunity to improve their skill in reading of operator 
interfaces. Teamwork and communication skills were also mentioned 
as positively impacted. 

3. How did the trainee rate: 

- the learning experience -- 4.2 on 1 to 5 scale (5=high) . 

- the assessment process -- used for diagnostic purposes only. 

- course usefulness -- 4.1 on 1 tec 5 scale. 

- course practicality -- high, all exercises were based on 
work examples. 

- instructor performance -- 4.3 on 1 to 5 scale* 

- course content -- 4.1 on 1 to 5 scale. 

- course activities — high, all exercises used workplace 
materials . 
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OBJECTIVE NUMBER TWO : IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES 
(Continued) 

4. Did the trainee feel (s)he was a partner in the learning 
process? The trainees provided some input into the course plan at 
the development stage and provide continuing criticism of the 
actual content and exercises as the course continued. The trainee 
comments were reflected in course revisions and exercises for the 
next sessions. 

5. To what extent were line workers basic skills increased? 
Increases in reading comprehension, teamwork, verbal and written 
communications, and problem solving abilities were reported. 

6. Were the learning objectives met? Objectives were met to 
varying degrees. Original objectives had to be changed often 
during the course due to changes in the work environment and due 
to schedule changes. Also, some personnel changes in mid-course 
required that some material had to be repeated instead of moving 
to more advanced topics . 

7. In a pre/posttest comparison, were there positive knowledge 
gains in basic reading, writing, and math skills? No pre and post 
tests were completed for this series of courses. The scheduling 
of participants was very sporadic due to changes in production 
requirements and participant availability. The participants were 
divided into three classes based on ability. The lowest ability 
level group included some non-readers. Very little improvement 
was reported from this group. The other two groups reported 
improvement in word recognition of technical words used on the 
operator interface panels. 

8. Did trainees perceive an acceleration in knowledge gains? 
No reports of any improvements in this area. 

9. Was the program long enough? No, the program needs to be 
longer to make it possible to show improvements with all 
learners . 

10. Did the trainees have ample time to integrate subject 
content, skills, and processes? No, more time would have been 
helpful . 



OBJECTVE NUMBER THREE: DESIGN AND ESTABLISH PROGRAM WHICH 
ENCOURAGES UTILIZATION . 

EVALUATION QUESTION: 

1. What are the average and typical frequencies of use? 
Attendance was very good. Forty of the 48 hours of class were 
offered in 8 -hour sessions as the fifth work day of the normal 
work week. The final 8 -hour session was conducted on a Saturday, 
as a sixth work day, with overtime pay. Scheduling made it very 
convenient for participants to attend. 

OBJECTIVE NUMBER FOUR: QUALIFICATION RATES FOR ADVANCEMENT 
EVALUATION QUESTIONS: 

1. To what extent has the Workplace Literacy Program provided 
opportunities for advancement? No advancement opportunities to 
report at the present time. 

2. To what extent has the program increased the qualification 
rate of job applicants? In this area, several marginal performers 
have shown increases in their ability to perform minimal job 
functions. They were in danger of being put on probation for poor 
performance. This has been avoided by their skill improvements 
due to the course. 

3. To what extent do employees feel a sense of empowerment, added 
responsibility, and increased decision making? Several trainees 
reported that they are making fewer calls to their supervisors 
for advice and direction when responding to minor process 
deviations . 

This seems to be a common result of this training program. 

OBJECTIVE NUMBER FIVE: ATTENDANCE 

1. What was the frequency of attendance? 

Attendance was very good, over 95%. Since attendance was 
mandatory 

as part of a normal work week, the only participants who did not 
attend were ill or on vacation. 

2 • Did anyone leave the program? 

Several participants left the program* due to schedule changes. 
All employees in the target work group were required to attend. 
Those who left the work group voluntarily were not allowed to 
attend further clashes. 

3. Why did trainees leave the program? 

Trainees left the work assignment for other positions within the 
manufacturing facility. The new positions were, more desirable for 
a variety of reasons work load, shift, or supervisor . 
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V. BUTTERF INQER PACKAGING SYSTEMS OPERATOR 

OBJECTIVE NUMBER ONE: CORRELATE WORKPLACE LITERACY SKILLS 
ENHANCEMENT WITH MANUFACTURING PERFORMANCE. 

EVALUATION QUESTIONS: 

1. What are the average and typical outcome measures of workplace 
skills enhancement ? 

a) What are average and typical student reaction outcomes? 
Participants rated the course 4.6 on a 1 to 5 scale (5=high) . 

b) What are average and typical learning outcomes? 

Math and problem solving improved, based on pre and post test 
scores . 

c) What are average and typical performance-based outcomes? 
Supervisors and managers reported improved job performance in 
ability to diagnose production problems and in completion of 
daily reports. 

2. What are key indicators of quality manufacturing performance? 
Production outputs, downtime, scrap and rework. 

3. What is the magnitude of the correlation of these measures? 
All areas have shown slight improvements since the commencement 
of this training program. Several other initiatives have also 
influenced these indicators of quality manufacturing performance. 
Supervisors reports clearly state that the participants have been 
responsible for improvements in these factors. 

OBJECTIVE NUMBER TWO; IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES 

EVALUATION QUESTIONS: 

1. Did the training positively impact trainee attitudes toward 
the learning process? Yes, trainees reported renewed interest in 
the learning process. Several have continued their formal 
learning with other coursework within the facility, external to 
the facility, and with computer-delivered lessons on their own 
time . 

2/ Did the trainee appreciate the learning experience? 
Very positive response to the availability of this program. 

3. How did the trainee rate: 

- the learning experience 4.6 on 1 to 5 scale (5=high) . x 

- the assessment process used for diagnosis, very well 
received . 

- course usefulness - 4.3 on 1 to 5 scale. 
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OBJECTIVE NUMBER TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 

TRAINEES 

(Continued) 

3. How did the trainee rate (Continued): 

- course practicality has helped with completion of job 
duties . 

- instructor performance — 4.8 on 1 to 5 scale. 

- course content 4.3 on 1 to 5 scale. 

- course activities some activities should have more time 
devoted to them. 

4. Did the trainee feel (s)he was a partner in the learning 
process? Some of the trainees were consulted as part of the 
original job task analysis and course development. During the 
course, each trainee was able to customize their individual 
assignments based on their abilities and needs. 

5. To what extent were line workers basic skills increased? 
Improvements in problem solving and math skills were measured. 
Interpersonal communications and teamwork were reported to be 
positively impacted. 

6. Were the learning objectives met? 

Instructors reported that they actually exceeded the objectives 
in several areas due to the strong skills possessed by most 
participants . 

7. In a pre/posttest comparison, were there positive knowledge 
gains in basic reading, writing, and math skills? 

Reading and math skills improved from 25 - 33% on average. No 
writing tests were given. Supervisors reported improved quality 
of daily production reports. 

8. Did trainees perceive an acceleration in knowledge gains? 
Yes, trainees especially reported improvements in problem-solving 
ability. 

9 . Was the program long enough? 

Yes. Additional work may be needed to bring skills to new levels, 
but the objectives were satisfied with the course as structured. 

10. Did the trainees have ample time to integrate subject 
content, skills, and processes? Yes, jthe course was conducted in 
2 -"hour segments, twice per week, for a total of 12 weeks. Time 
was reported to be adequate. 




OBJECTIVE NUMBER* THREE: DESIGN AND ESTABLISH PROGRAM WHICH 
ENCOURAGES UTILIZATION, 

EVALUATION QUESTIONS: 

1. What are the average and typical frequencies of use? 
The course was offered to a total of 36 employees; 19 enrolled in 
the course. This was typical of all six targeted groups for this 
project . 

OBJECTIVE NUMBER FOUR: QUALIFICATION RATES FOR ADVANCEMENT 
EVALUATION QUESTIONS: 

1 . To what extent has the Workplace Literacy Program provided 
opportunities for advancement? At this time, there have been no 
advancement opportunities available for the participants. 

2. To what extent has the program increased the qualification 
rate of job applicants? All participants were well-qualified for 
their positions. The program helped improve their daily report 
writing and scrap, rework, and production rates. None of the 
participants was in need of training to meet minimum 
qualifications . 

3. To what extent do employees feel a sense of empowerment, added 
responsibility, and increased decision making? Supervisors 
reported fewer calls for advice on minor machine problems. 
Participants were seen interacting more with the mechanical and 
electrical repairers working on their production lines. 

OBJECTIVE NUMBER FIVE: ATTENDANCE 

EVALUATION QUESTIONS: 

1 . What was the frequency of attendance? 

A ;tendance was very good (over 90%) ; the only absences reported 
were for conflicts with overtime required for production purposes 
and occasional problems with transportation. 

2 . Did anyone leave the program? One person left the program due 
to escessive absenteeism. 

3 . Why did trainees leave the program^ 

The one person who left had trouble arranging transportation 
home. The course was offered immediately after normal work hours 
for a 2 -hour period. The trainee normally rode home with a person 
who was not in the class. 
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VI. BUTTERFINGER MANUFACTURING SYSTEMS OPERATOR 

OBJECTIVE NUMBER ONE; CORRELATE WORKPLACE LITERACY SKILLS 
ENHANCEMENT WITH MANUFACTURING PERFORMANCE . 

EVALUATION QUESTIONS: 

1. What are the average and typical outcome measures of workplace 
skills enhancement? 

a) What are average and typical student reaction outcomes? 
The overall course was rated 4.8 on a 1 to 5 scale (5 high) . 

b) What are average and typical learning outcomes? 
Learning outcomes did not show any change in performance. 

c) What are average and typical performance-based outcomes? 
Supervisors reported improved performance on production start 
ups, more accurate completion of daily reports, and a reduction 
in calls to supervisors for advice on minor problems. 

2. What are key indicators of quality manufacturing performance? 
Scrap, rework, production output. 

3. What is the magnitude of the correlation of these measures? 
All three have shown slight improvements based on supervisors 
observations. None of the improvements have been analyzed 
statistically. 

OBJECTIVE NUMBER TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES 

1. Did the training positively impact trainee attitudes toward 
the learning process? Yes, trainees demonstrated increased 
commitment and interest in improving their skills. 

2. Did the trainee appreciate the learning experience? 
Trainees were very appreciative of the opportunity to participate 
in this course. 

3. How did the trainee rate: 

- the learning experience 4.8 on 1 to 5 scale (5=high) . 

- the assessment process used for diagnosis only, no 
feedback . 

■ : - course usefulness 4.7 on 1 to 5 scale* 

- course practicality participants report using what they 
learned on a daily basis. 

- instructor performance 4.8 on 1 to 5 scale* 

- course content 4.7 on 1 to 5 scale. 

- course activities trainees reported that exercises and 
other activities were interesting and useful. 
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OBJECTIVE NUMBER' TWO: IMPROVE WORKPLACE LITERACY SKILLS OF 
TRAINEES ( Cont inued) 

4, Did the trainee feel (s)he was a partner in the learning 
process? Yes, trainees helped design the original curriculum and 
provided direction for individual modifications as the course 
progressed . 

5, To what extent were line workers basic skills increased? 
Math, reading, and problem-solving skills all showed improvement. 



6. Were the learning objectives met? 
Yes, all objectives were met. 

7. In a pre/posttest comparison, were there positive knowledge 
gains in basic reading, writing, and math skills? Due to the 
small size of the classes (10 total between 3 classes) , and wide 
range of abilities, pre- and post-tests were not given for this 
course . 

8. Did trainees perceive an acceleration in knowledge gains? 
Trainees reported that their abilities increased, even if not 
able to support this perception by improved test scores. 

9. Was the program long enough? Trainees would like to spend more 
time on problem solving activities. These seemed to be useful to 
them on their jobs. Detailed analysis of specific work place 
problems was suggested for additional courses . 

10. Did the trainees have ample time to integrate subject 
content, skills, and processes? Course was conducted in 2 -hour 
segments, twice weekly, over a 12 week period. This appeared to 
be a good time frame for this type of material, 

OBJECTVE NUMBER THREE: DESIGN AND ESTABLISH PROGRAM WHICH 
ENCOURAGES UTILIZATION. 

EVALUATION QUESTIONS: 

1. What are the average and typical frequencies of use? 
Of a total of 12 potential participants, 8 attended. Two 
additional attendees were from a support group who expressed 
interest in participating. They were allowed to participate based 
on" the terms of the grant that required all interested learners 
be given the opportunity to attend • 
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OBJECTIVE NUMBER FOUR; QUALIFICATION RATES FOR ADVANCEMENT 



EVALUATION QUESTIONS: 

1 . To what extent has the Workplace Literacy Program provided 
opportunities for advancement? At this time, no opportunities for 
advancement have been available. 

2. To what extent has the program increased the qualification 
rate of job applicants? All participants have been in their 
current positions for extended periods and are performing 
acceptably. 

3. To what extent do employees feel a sense of empowerment, added 
responsibility, and increased decision making? Supervisors report 
fewer calls for advice and direction on routine production 
deviations. Also, increased communication between participants 
has been observed. 

OBJECTIVE NUMBER FIVE: ATTENDANCE 
EVALUATION QUESTIONS: 

1 . What was the frequency of attendance? 
Attendance was very good, in excess of 90%. 

2 . Did anyone leave the program? 

No. One participant had poor attendance, but did not officially 
leave the program. 

3 . Why did trainees leave the program? 
None left. 
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ABSTRACT 

The following is a critical review of the Internal 
Evaluation of Nestle 's Federal Workplace Literacy Project. This 
short paper is divided into three sections. The first 
establishes the basis for evaluation review or meta-evaluation 
and gives those models with differing criteria of goodness in 
evaluation. The second section looks at the structure of the 
evaluation project as a whole. We made some suggestions for 
reformatting the contents and enhancing readability. In section 
three we make specific comments on the draft itself. Finally, we 
close by endorsing the evaluation of what appears to be an 
excellent educational program. 
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In preparation for this commentary , or meta-evaluation, of 
the Nestle' s Internal Report of its Workplace Literacy Grant, 
PR/Award No. U198A30096, I have read the report and accompanying 
supporting documentation. On the whole, the evaluation questions 
seem to be answered and the program effective in supporting its 
goals. The overall design for my comments are divided into three 
major sections. The first section deals with the proper conduct 
of meta-evaluation, for which we have consulted several cited 
sources. In this section we provide the standards and criteria 
for successful evaluation. In the second section we consider how 
best to structure the report in terms of its groupings of 
supporting evidence. The third and final section poses 
questions and seeks elaborations from the authors of the report. 

Section I: Why Meta-Evaluation? 

The primary assumption of meta-evaluation is that evaluation 
should be subject to critical review and scrutiny by an outside 
source. It should provide a balance of perspectives to aid 
decision making. We looked at three models of meta-evaluation. 
Cook and Gruder (1978), St. Pierre (1982), and Martin (1981) have 
identified several models for meta-ev^luation and specified 
criteria to judge the adequacy of evaluation* Cook and Gruber 
(1982) were among the first to propose a review essay or 
narrative of evaluations. What other proponents of meta- 
evaluation suggest include secondary reanalysis of data and 
reviews of the literature* 
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Rather than reanalyze data or conduct extensive literature 
reviews, we opt for the review essay because of its practicality. 
Stake (1969) proposes that all evaluations have some sort of 
criteria and/or structure that separates good evaluations from 
bad. In his influential article, "Table of Contents for a Final 
Evaluation Report" he proposed the following as hallmarks of 
good evaluations: 

o Objectives of the Evaluation 

o Specifications of the Program 

o Program Outcomes 

o Relationships and Indicators 

o Judgment of Worth 

While these categories are generic and broad enough to be 
generalizable to most any evaluation, they lack specificity to 
make meaningful judgments concerning specific evaluation. 

Martin (1981), working for the California Evaluation 
Services, has developed a checklist of criteria which usually 
indicative of quality evaluations. As all criteria are not 
germane to this evaluation, some have been eliminated. We use 
these as criteria for use in our meta-evaluation. 

Overall Proposal 

o Organization 

o Identification of Need 
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o Definition of Goals, Objectives and Intents 
o Readability 

Evaluation Design 

o Needs Assessment Data 

o Stated Purpose of the Evaluation 

o Description of Instruments 

Assessment Techniques 

c Testing Schedule 

o Validity/Reliability of Instruments 

o Data Storage and Retrieval 

Report Characteristics 

c Adequacy of Recommendations 

o Generalizability of Results 

o Continuation of Program 

These criteria are hallmarks of good evaluations but they do 
not have to be presented in this specific format. Instead, much 
of what is presented is essentially covered in the standard 
research report format which we urge Nestle 's to adopt in their 
internal evaluation . 

Section II: How to Structure the Evaluation Report 

In the Initial Draft of the Internal Evaluation, the 
majority of the report was devoted to the answering the 
evaluation questions posed in the evaluation design in a very 
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straight-forward way. While we applaud the fidelity to the 
design, we urge the more traditional evaluation research format. 
These include Executive Summary, Background or Rationale of the 
Program, Subjects, Program, Teaching Staff, Results, and 
Conclusion . 

Further, the report contained no graphic displays nor charts 
which would be helpful for the reader to comprehend large amounts 
of data. Nor did the report switch fonts between evaluation 
questions and responses to those questions. If it is still 
feasible at this point, the company nay want to consider 
incorporating graphs or charts that add visual summary to the 
data. Also, to enhance the visual appeal of the report, consider 
offsetting the design questions from the answers by switching 
fonts, underlining or italics. Finally, I find a ragged right 
page column easier to read than right justified. This eliminates 
the different spacings between the words and letters. These 
suggestions should be considered subjective. 

It appears that there is ample data to support convincing 
arguments that these programs were, in the large part, 
successful. What is necessary is to rearrange the material into 
a standard evaluation format. These include: 

Executive Summary 

This is a one to two page summary of the program, its 
rationale, the sample population and the results of the 
evaluation. This section would also contain recommendations or 
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these could be put separately under its own heading. 
Rationale for the Program 

This section provides the rationale for the program or what 
need it addresses. It tells why the program is necessary. It 
also 

tells what goals the program hope to address? It provides the 
answer to what this program and not another. 

Sample 

This sections would give a combined portrait of the students 
in the population. It gives the total number of participants, 
why the entered the program. It reports what entry skills the 
program participants have and why they entered the program. 
Finally, it presents the racial, ethnic and gender breakdown of 
the participants. 

Teaching Staff 

Sometimes differences in program outcomes are attributable 
to differences in teaching quality. Although this does not seem 
to be problem in this evaluation, you 'may want to consider make 
this point more emphatic. 

Instrumentation 

Unless it is not specifically called for, we recommend that 
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all the instruments used to evaluated the courses be included in 
appendices. Alternatively, the company may want to how the 
instruments were designed, how issues of validity and reliability 
were dealt with. You may want to include how the instruments 
were piloted, either among colleagues or among students. 



Results 

This section presents the results of the six courses along 
the four dimensions posed by the evaluation design. For example, 
graphing the satisfaction and achievement results of the six 
courses (one course did not collect these data) would permit 
readers to see at a glance how each course compares to each 
other. These data displays offer the reader to compare each 
course to the other but also against the five point criteria. 

In addition, it appears that there was a considerable amount 
of interview data. We think it would strengthen the results to 
have these more available in the report. Actual quotes from the 
sources are best. 

Section III: Specific Comments on the -Draft 
Fork Lift Loader /Checker 

Despite the difficulties of being a pilot class for the 
project, it appears that this course had very positive results. 
There were positive gains in reading, math and writing. It also 
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appears that the students enjoyed the program and the instructor. 
Finally , supervisors noticed a change subsequent to training. 

There appears to be only one difficulty that is not fully 
explained in the draft. With a target audience of eight , only 
three chose to opt into the program. There could be several 
factors and we assume that scheduling and the fact that it was a 
pilot program rank high. The company may want to explain more 
fully why less than 50% of the target audience did not choose to 
attend . 

Pearson Kitchen Helpers 
BB's Processing Systems Operator 
Butterfinger Manufacturing Systems Operator 
Butterfinger Packaging Systems Operator 

The above four courses are similar enough to be treated as a 
group. On first read of the evaluations , it appears that these 
courses were very successful and we are sure that they were 
because they produced favorable ratings , satisfactory achievement 
gains and good job performance transfers. But there are some 
questions that need to be addressed to ensure that this is in 
fact the case. First , it is difficult to determine how many 
students actually enrolled in the course? And why other eligible 
students did not enroll. Also, the following questions should be 
addressed here if they were not included elsewhere in the report: 



How was the course structured? 

Did the structure of the course differ substantially from 
the others? 

Nestle 's may want to consider including the answers to these 
questions in their final report. 

Summary 

It appears that the Nestle 's program has produced 
considerable success in promoting literacy in the workplace. The 
courses were solidly designed with high participant input. The 
instructors were very competent and received high ratings from 
their students. Where measured, the courses produced excellent 
results in terms of student satisfaction, student achievement, 
and skills transfer to the workplace. 

Our recommendations for the report are the following. 

o Restructure the format to be more consistent with a 
traditional research and evaluation format. 

o If possible, incorporate graphics. 

o If possible, use varying fonts offsetting questions 

from response, 
o Provide instruments in appendices. 
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